Aiming at the low segmentation precision caused by the lack of image feature extraction ability of the existing fractional-order nonlinear models， an instance segmentation model based on fractional-order network and Reinforcement Learning （RL） was proposed to generate high-quality contour curves of target instances in the image. The model consists of two layers of modules： 1） the first layer was a two-dimensional fractional-order nonlinear network in which the chaotic synchronization method was mainly utilized to obtain the basic characteristics of the pixels in the image， and the preliminary segmentation result of the image was acquired through the coupling and connection according to the similarity among the pixels； 2） the second layer was to establish instance segmentation as a Markov Decision Process （MDP） based on the idea of RL， and the action-state pairs， reward functions and strategies during the modeling process were designed to extract the region structure and category information of the image. Finally， the pixel features and preliminary segmentation result of the image obtained from the first layer were combined with the region structure and category information obtained from the second layer for instance segmentation. Experimental results on datasets Pascal VOC2007 and Pascal VOC2012 show that compared with the existing fractional-order nonlinear models， the proposed model has the Average Precision （AP） improved by at least 15 percentage points， verifying that the sequential decision-based instance segmentation model not only can obtain the class information of the target objects in the image， but also further enhance the ability to extract contour details and fine-grained information of the image.
